Second order conditions for optimality 
and local controllability of discrete-time systems 
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Abstract 

We study local controllability and optimal control problems for invertible discrete-time 
systems. We present second order necessary and sufficient conditions for optimality and for 
local controllability. The conditions are stated in geometric terms, using vector fields naturally 
associated to the system. The Hessian of the optimal problem is computed in terms of Lie 
brackets of vector fields of the system. 
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1 Introduction 



Nonlinear discrete-time control systems 

E: x(t) = f{x{t - l),u{t)) 

are much less understood, comparing to continuous-time systems. This is due to the fact that 
algebraic and geometric tools available in the continuous-time case are not present or, at least, 
have not been used much for their analysis. Here we have in mind vector fields, Lie bracket and 
Lie algebraic techniques which are very helpful in the theory of continuous-time systems. 

There is a class of discrete-time systems where such tools are available, however. These are 
invertible systems, where the state equations are solvable backwards in time, that is x(t — 1) is 
uniquely defined for given x(t) and u(t). Families of vector fields can be assigned to such systems 
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(see [HI [10] ) and can be used for analyzing their general controllability properties [HED]. Invertible 
systems appear e.g. in modeling of continuous-time systems, as in the case of sampling, and in 
numerical schemes approximating continuous-time systems. Codistributions can be also assigned 
to such systems and have been useful for characterizing their observability [3] . Recently, differential 
geometric tools have been used to analyze the accessibility of implicit discrete-time systems [T3] . 

For continuous-time optimality of a trajectory is closely related to its local controllability prop- 
erty. Namely, for most optimal control problems, optimal trajectories lie on the boundary of the 
reachable set. On the other hand, it is a hard problem to find criteria in terms of the system vector 
fields which characterize local controllability along a trajectory (i.e., when the trajectory lies in 
the interior of the reachable set). Deep and far reaching results in this direction were obtained 
in the last forty years (see e.g. [5j EJ [TTJ [15]) but a complete characterization seems beyond the 
reach (HE]. General second order conditions for optimality in continuous-time can be found in [2J. 

In this paper we analyze invertible discrete-time systems using the formalism of vector fields 
from [TO]. We present (Section [3]) second order sufficient and necessary conditions for local control- 
lability in terms of those vector fields and their Lie brackets. In Section [5] we present basic lemmata 
which are then used for proving the sufficiency result in Section [3j Second order optimality con- 
ditions are presented and proved in Sections El and [TJ using the same lemmata. The proof of one 
version of these results is based on a geometric lemma in [2J on local openness of a nonlinear map 
at a singular point. We include an illustrating example in Section [H 

Our infinitesimal analysis of local controllability, with the use of Lie bracket, can be considered as 
a starting point to identifying higher order sufficient or necessary conditions for local controllability 
of discrete-time systems (analogous to conditions in the continuous-time case), in particular to 
identifying so called "bad brackets" (see El Ell E2J ES] for the case of continuous-time). 

2 Preliminaries 

Let M and U be two sets called state space and control space, respectively. A map / : M x U — )■ M 
defines a nonlinear discrete-time control system with the dynamics 

E: x(t) = f(x(t-l),u(t)), 

where x and u are called state and control, respectively, and t G Z. 

We will assume that: 

(Al) M is an open subset of MJ 1 or a smooth differentiable manifold of dimension n; 

(A2) U is a subset of M m , with nonempty interior, and the closure of the interior of U contains U ; 

(A3) the map /: M x U -»■ M is of class C 2 ; 

(A4) the system is invertible, which means that the maps f u : M — > M, u e {/ ', are diffeomorphisms 
onto open images. 
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Condition (A3) means that / has a C 2 extension to M x U, where U C K m is an open superset 
of U. Above, and in the sequel, we denote 

fu(x) = f(x,u). 

The map f u :M—tM defines the one-step transition defined by control u. The invertibility 
property (A4) means that f u (M) C M is open and each f u : M — > f(M) is a C 2 diffeomorphism. 

Assumption (A4) is needed for associating natural vector fields to system S. Sometimes we will 
assume that the system is strongly invertible, which means that the maps f u : M — > M, u £ U, are 
diffeomorphisms onto M. 

It will be convenient to use the notation t = i and write the system equations in the form 

S: Xi = f Ui (xi^i), where x { = x(i), u { = u(i). 

Given an initial state x and a control sequence Ui, . . . ,un, the trajectory of E is defined by the 
sequence of states xi, . . . ,x^, where Xi is given by the composition of maps f Ui o • • • o f U2 o f Ul 
applied to x . We will usually omit the composition sign and write = f Ui ■ ■ ■ f Ul (x ). The set of 
points reachable from xq in iV forward steps is denoted by 

TZ + (x ,N) = {x e M | 3 (m, . . . ,u N ) £ U N such that x = f UN . . . f Ul {xo)}- 

Given a subset U C U N of control sequences (ui, . . . , Wjv), we define the corresponding reachable 
set 

Tl + (x , U) = {x £ M | 3 (m, . . .,u N ) £ U such that x = f UN . . . f ui (x )}. 

With the purpose to discuss local controllability of discrete-time systems we will provide an 
infinitesimal description of local deformations of admissible trajectories of S. We will use vector 
fields introduced in [10] (see also [El [8] for earlier work), where they were needed for characterizing 
accessibility and controllability. 

Assume first that the control is scalar, that it is U C KL Then the following vector fields 
depending on u can be associated to the invertible system S (cf. [10J ) , 



X « +(X) = I 
d 

X « (X) = We 



f-'of^x), Y u + (x)=§- 

_ d 

fuof u+e {x), Y u (x) = — 

e=0 0€ 



e=0 



fulofu(x), (1) 

i 

fu+eOf-\x). (2) 



e=0 



The vector fields in ([T]) can be used for the infinitesimal analysis of the variations of forward 
trajectories of S, whereas the vector fields in fl2]) play an analogous role for backward trajectories. 
Note that A+ and are well defined for all x £ M, if £ is invertible (for small e the point f u+t (x) 
is in the domain of f~ x , by the implicit function theorem), while the same holds for X~ and Y~ 
only when E is strongly invertible. Due to assumptions (A2) and (A3), these vector fields are well 
defined for any u £ U and are of class C 1 . 
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We will mainly use the w-depending vector fields X+, however similar results can be obtained to 
describe local controllability and optimality by means of the vector fields in when reachability 
is defined by backward trajectories. The vector fields can be alternatively defined as 

x+(x) = (df u ( X )r i -^f u (x). 

In the case of multidimensional control these w-depending vector fields depend, additionally, on the 
index r of the component of u = (u 1 , . . . , u m ) G U, 



fu 1 O f u+eer (x) 

e=0 



where e r is the r-th versor in R m . 

Given a vector field Y and a control u G U, we define another vector field using the diffeomor- 
phism f u on M, 

(Ad u Y)(x) = (d/ u (x))- 1 F(/ u (x)), 

where df u (x) is the differential of f u evaluated at the point x. (The above definition of Ad, used 
throughout the paper, is convenient for analyzing forward trajectories. Note that it does not 
match the usual notation of Lie group theory adopted to left actions.) More generally, denoting 
fu k ...ui( x ) — fu k ■ ■ ■ fui{x) we define the following vector fields 

(Ad Uk .. Ml Y)(x) = (Ad Ul ---Ad Uk Y)(x) = u\f,„ .(,•)) i Y(f„ i ..,:(-r)). 

Note that, in the case of scalar control, 

d 



(Ad Uk ... ul x: o )( X ) = - 

For multidimensional control 

(Ad Uk „ Ml X U(hr )(x) = — 

where e r is the r-th versor in R m . 



fu k ...u 1 fu fu +efu k ...u 1 (x)- 



e=0 



f'u k ...u\fuo fuo+ee r fuk...ui{x), 



e=0 



Proposition 1. JTflt Proposition 3.2] For scalar control the following equalities hold for each u G U : 

(1) x+ = -r+ ; x- = -y-, 

(2) X+ = -Ad u X-, Y+ = -Ad u Y-. 

For multidimensional control analogous equalities hold for the vector fields X+ r , Y+ r , X~ r and Y~ T . 

For so defined vector fields we can compute their Lie bracket [■, ■] in the usual way. We also 
denote the Lie bracket of Y and Z as adY(Z) = [Y, Z\. 

Proposition 2. fjQ|. Proposition 3.3] The following equalities hold for any vector field Z and any 
scalar u G U : 

— Ad u Z = adX+(Ad„Z), q^^Z = adX-(Ad- 1 Z). 
In the multidimensional control case these equalities take the form 

-Ad u Z = adX+ (Ad u Z), t-Mu'Z = adX~ (Ad; 1 ^). 
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3 Local controllability and geometric optimality 



We first assume that the control is scalar, u E U C R. Consider an admissible control sequence 
u = (ui, . . . , mat). Given an initial point x , we say that system E is N-step locally controllable from 
x along the trajectory x(t) corresponding to u (shortly, (xq,u)- locally controllable) if 

x(N) G mtn + (x ,N). 

The system E will be called strongly (xq,u) -locally controllable if for any neighborhood U C U N of 
u we have 

x(N) emtn + (x ,U). (3) 

Then E has this property, with the same xo, for any control sequence u of length N' > N with u 
being its initial part. 

In order to state a sufficient condition for local controllability we need the following notation. 
For a fixed control sequence u = (ui, . . . ,%) we introduce the first variation vector fields 

Yi = Y^.. Ui := Ad Ul ■ ■ ■ Ad Ui _,X+ , i = 1, . . . , N 

(in particular, = X+). Note that they have the recurrence property 

Y i - AH Y l ~ l 

1 U 1 ---U i ^^Ml 1 U 2 -Ui- 

For a given x we introduce the space of vectors in T X M 

L fi (x)=spaii{yS(a;), i = l,...,iV}. (4) 

The family of vector fields defining L a (x) does not necessarily describe a minimal set of generators 
for such a subspace. We define a subspace, called the kernel, of the vector space of coefficients 
a — (ai, . . . , Oat), 



i=i 

For i,j G {1, . . . , A^} we define the second variation vector fields 

zs = zf = \m,Yi}, i<j, (5> 

where the square bracket denotes the Lie bracket of vector fields on M. Equivalently, are given 
by 

Z^ = Zi* = ^ Ad Ul ■ ■ ■ Ad Ui _, [X+, Ad tti • • • Ad u ._ 1 X+], i < j. 
Given a point rr G M, consider the vector-valued quadratic form on the space of parameters 

N 

H n (a) = H(a) = ^ aiajZ%(x ) (7) 



(the subscript u will be omitted). If A is a covector in T* Q M, the formula A H(a) = ^ • aidj\(Z l J (xq)) 
defines a real-valued quadratic form. For a quadratic form Q on a finite dimensional real vector 
space V we denote by Ind + Q (resp. lnd~Q) the maximal dimension of a subspace W C V such 
that Q restricted to W is strictly positive definite (resp. strictly negative definite). Recall also that 
Q is indefinite if there are vectors v and w such that Q(v) > and Q(w) < 0. 

Given a subspace L C T X M we denote by L 1 - its annihilator, L 1 = {A 6 T*M : A|l = 0}. We 
will often use a covector A G T* Q M annihilating all vector fields Y£ at x , i.e., 

Ays(x ) = o, i = i,...,jv. 

This condition is shortly written as A G (L u (xo))' L - 

Theorem 3. Assume that S satisfies (Al)-(A4)- Given a fixed initial point Xq G M, consider an 
admissible control sequence u = (ui,...,u^r) such that u G (mtU) N and let k = codimL s (xo). 
Then the following statements hold. 

(a) If k = 1 and XH restricted to K q (xq) is indefinite, for any A G (L u {xq)) ± , A 7^ 0, then system 
S is strongly (xq,u) -locally controllable. 

(b ) In general, if S satisfies the condition 

Ind~(A H)\ Ku(xo) > k, V A G (L^xo)) 1 , A ^ 0, (8) 
then it is strongly (x ,u) -locally controllable. 

Note that replacing A with —A gives the same inequality for Ind + . Thus condition (jSJ) says that 
the quadratic form A if on M. N , defined by the second variation vectors at Xq, has at least k positive 
and k negative "eigenvalues", when restricted to the subspace K a (xo) C M. N . 

We will now state a similar result for the multidimensional control, i.e., for U C M m . We fix 
a control sequence u = (ui, . . . ,u^) G U N , where Ui = (uj, . . . , u™). Analogously to the scalar 
control case we define first variation vector fields 

= Ad Ul • • • Ad Uj _ 1 X 1ti)r , 

for % — 1, . . . , N, r — 1, . . . , m. Given a point x we introduce the space of vectors in T X M 

L u {x) = span{Y7(x), i = 1, . . . , N, r = 1, . . . , m}. 

Consider the space of coefficients a — (ai, . . . , ajv), where = (a], . . . , a™ - ). We will use its subspace 

{N m ~\ 
i=l r=l ) 

called the kernel. For i,j G {1, . . . , N}, r, s G {1, . . . ,777.}, we define second variation vector fields: 
Zu" s = = \ [Y7, Yt\ = \ Ad Ul • • • Ad Wi _ x [X+ tr , Ad Ui ■ ■ ■ M^XtJ , i < 3, 
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2?rr,is _ . . . Ad u . _ i __X lt . s . 

For xq € M consider the following vector-valued quadratic form on R- 



H(a) 



(x ). 



(9) 



r,s=l, 



.iV 

.,771 



With the above definitions we have the following result. 



Theorem 4. Theorem^ remains valid in the case of multidimensional control. 



The following converse result will be proved in Section E] using Theorem [T2 
Theorem 5. If there exists A G (L^xo)) 1 " such that \ H\k u { X0 ) is positive definite, 

i/ien S is noi strongly (x ,u) -locally controllable. 

Theorems [3] and H] can be used for obtaining necessary conditions on geometric optimality. Recall 
that, given an initial point xo and control sequence u = (ui, . . . , ttjv), the corresponding trajectory 
x(i), i = 0, . . . , N, of system £ is called geometrically optimal if it lies on the boundary of the 
reachable set, i.e., 

x(i) e dTZ + (x ,i), i = l,...,N. 

It follows from the invertibility of the system that condition x(j) G int7?. + (xo, j) implies the in- 
clusion x(k) G int7?. + (xo, k), for any k > j. Thus, the above condition for geometric optimality is 
equivalent to x(N) G &R + (xq, N). Theorems [3] and H] trivially imply 

Theorem 6. If the trajectory of system S corresponding to initial point xq G M and an admissible 
control sequence u = (u\, . . . ,un) such that u G (int U) is geometrically optimal, then there exists 
a nonzero covector A G (^(xo)) -1 such that 

hxT(A H)\ Kc (xo) < codimL s (x ). 

In particular, if L u (xq) is the whole tangent space then the trajectory is not geometrically optimal. 
If codimL s (xo) = 1 then it is necessary for geometric optimality that there exists a nonzero A G 
(L s (xo)) _L (unique up to a positive multiplier) such that the quadratic form A H restricted to the 
kernel K u (xq) is non-negative definite. 
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4 Example 



Take M = M 3 and let U C R be an open subset. Consider the discrete-time control system on M 

,2< 



fu(x,y,z) 



( u \ 



xz — y 
,2 



V 



2 + 



(10) 



/ 



Denote the state x = (2,?/, z). As the state is 3-dimensional and the control is scalar, the system 
is never locally controllable in two steps. It is 3-steps (x, w)-locally controllable if Y^,Y?,Y? are 
linearly independent at x (this simply follows from the inverse function theorem). If they are not 
linearly independent, the index condition from Theorem |3] does not help as it never holds here for 
3-steps controls. The reader may analyze the cases where Theorem H] is applicable. 

We will consider 4-steps controls u = (ui, u 2 , u 3 , w 4 ). Given an initial condition x = (x,y,z), 
we have 



(-1 l\ 

d/ u (x) = z -1 x , 

V V 

The first variation vector fields at x are 



d/- X (x) 



(-10 1 \ 

— z —1 X + z 
\o 1 ) 



du 



(u\ 


w 



^(x) 



^! 1 (x)=X+(x) 
^ lU2 ( x ) = Ad Ul K U2 (x 

d/- 1 (x)y 1 i(/ ttl (x)) = 



M 

X 



Ml, 



/ 1 \ 

2x — - w? 

o 1 



«2, 



V 1 7 



^uU a U3 ( X ) ~~ ^^"1 Yu 2 U 3 ( X ) 



/ 



d/- 1 1 (x)F tt 2 2U3 (/ Ul (x)) 







\ 



3x — 2z -\ — itn — it? 
2 



V 



W3, 



/ 



K-ffxl 



x) = Ad Ul Y % 



U2U3U4 ' 



d/-W4^(.U x )) 



1 
2 



U4. 
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The second variation vector fields can be computed using the definition 

M 



Zfix) 



x 

W 

( 1 \ 

2x - -u{ 
V 1 / 



3x — 2z — u\ + ^-u\ 



\ 
1 

2 

1 

/ 1 \ 

l.r - ~u\ + U 2 2 - -u\ 
1 



and the definition (j^J): Z % 1 = Z^ 1 = \\Yi, Y£], for i < j. Computing the Lie brackets we find that 



Zf{x) = - 



-1 

V / 
M 

1 

w 



11x112, ^i 3 (x) 



u 2 u 3 , Zl 4 (x 



-2 

V / 

2 

w 



M 1 M 3, ^s 4 ( X ) 



Z\ 4 (x 



1 

2 

1 
2 



/(A 

-1 

\ / 
(A 
-1 

0/ 



UIU4, 



u 3 u 4 . 



'ID 



Case I. Assume that 

tti 7^ 0, u 2 7^ 0, u 3 7^ 0, u 4 7^ 0. 

Then the dimension of L„(x) = span {Y^, Y?, Y?, Y£} is 3 (then £ is locally controllable) or it is 
equal to 2, if 

1 2 ^ 2 1 2 1 2 1 2 /io\ 

x = -^u 3 - -u 2 , z = -u 3 - -u 2 - ~u x . (12) 

Let x = (x,y,z) satisfy condition f[T2"j) . Then codimL^x) = 1 and the annihilating space L^(x) is 
generated by the covector 

A = 



The kernel is 



-x + -u 1 ,l,-x ] . 



Ui 



(13) 



u 2 



K u (x.) = {(ax, o 2 , a 3 , a 4 ) : a 3 = a 1; a 4 = a 2 > C 

U 3 Ui 



For A defined by ( TT3|) the quadratic form Aif is given by 

T 



XH{a) 



fai\ 

«2 

\a 4 / 



/ 





—U\U2 


— 2mim 3 


— MlM 4 \ 




/aA 




—U1U2 





M2M3 


2li2"U 4 




a 2 




-2u\U 3 


U2U3 




-M 3 M 4 




a 3 


\ 


—U1U4 


2ti 2 w 4 


-M 3 M 4 






\a 4 / 
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where 



1 . 2 1 2 

2x — 2z — u 1 + -u 2 , 



1 9 1 9 

- B = 2x + u\ ui. 

2 1 2 3 



Assuming ffTTj) . we have A = = B at points x G M satisfying condition (JEJ), where dim L fi (x) = 2. 
Then 



Aif(a) 





MA 


/o 


-1 


-2 -1\ 


MA 


1 


^2 


-1 





1 2 


^2 


2 


^3 


-2 


i 


-1 


^3 




w 


V-i 


2 


-1 0/ 


W 



where A- 



From the form of the kernel K u we see that, for fixed u satisfying ( TlT]) . the vector (01,02,03,04) 
is in the kernel if and only if A\ + A3 = and A 2 + A4 = 0. Therefore, taking A3 = —A\ and 
A4 = —A 2 we get 

(Atf| x J(o)= 4^-^-^). 

This expression is indefinite, treated as a quadratic function of the vector (01,02) parameterizing 
the kernel K a , which means that there exist values of ai and 02 such that the quadratic form is 
positive and negative. Thus, by Theorem [3j for any 4-step control u satisfying ffTT]) and any initial 
point fulfilling ( 1T2"]) the system is strongly (x, u)-locally controllable, even if f[T2"j) means that it does 
not satisfy the first order sufficient condition for local controllability. 

Case Ila. For other 4-steps controls, not satisfying fllip . we consider only one case where u± = 
U3 = and u 2 7^ 0, U4 7^ 0. In this case dim L«(x) = 1, if the initial condition satisfies x = —-^^ 

and dim Lu(x) = 2, otherwise. If x 7^ — we have k = codim L„(x) = 1. Then (L Q ('x)) ± 
is generated by the covector A = (—1,0,1), the kernel i^«(x) consists of vectors (ai, 0, a 3 , 0) T , 
Oi, a 3 G K., and the quadratic form A H evaluated on vectors in K Q ('x) is 



XH(a) 



2 1 2 
a 1 + a 3 . 



We see that XH restricted to Ku(x) is positive definite. It then follows from Theorem |5] that the 
system is not strongly (x, ^-locally controllable in this case. 

Case lib. If U\ = u 3 — 0, u 2 7^ 0, w 4 7^ and x = Theorem can be used again. Namely, 

the annihilator L^(x) is spanned by the covectors Ai = (—1, 0, 1), A2 = (2x, —2, 2x) and A G L^(x) 
has the general form 

A = aAi + 6A2 = (—0 
The corresponding quadratic form is 



2bx, -2b,a + 2bx). 



XH 



fa \ 

-2b 

a + b(Az + u 2 2 ) 

\0 -2b / 



The kernel i^u(x) is spanned by the vectors 

Vi = (1, 0, 0, of, v 2 = (0, 0, 1, 0) T , V 3 = (0, 1, 0, -1) 
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In the basis V\, V2, V3 the matrix {A H(Vi, Vj)} of the quadratic form XH\ K -^ is 

A#V ( X ) = diag{a, a + b{Az + uij), 46}. 

It is positive definite if a ^> b > 0. Using Theorem [5] we see that also in this case the system is not 
strongly (x, ^-locally controllable. 

Conclusions, (a) For any 4-steps control u satisfying fill I) (thus, for any N-steps control with 
initial part u) and any initial condition x the system fllUp is strongly (x, -u)-locally controllable, 
(b) If u satisfies u\ = W3 = and ui 7^ 7^ 114 then ffTO]) is not strongly (x, ^-locally controllable. 



5 Proof of local controllability 

We will prove the sufficiency result in Theorem [31 only (the proof of Theorem H] is analogous) . The 
proof, as well as further optimality results, are based on the following lemmata. 

Given a control sequence u = (ui, . . . , «jv) we define the composed map fu : M — > M, 

fu fujsf ' ' ' fui j 

which is a diffeomorphism. Let W — int U x • • • x int U C and consider the map F : W — > M. n 
defined for a fixed xq G M by 

F(u) = U(x ), (14) 

Lemma 7. We have 

lmdF(u) = df a (xo)Lu(x ). 
Lemma 8. The kernel of dF(u) is given by 

ker dF{u) = K Q (x ). 

Lemma 9. The second differential of F at u, restricted to the kernel ker dF(u) = K u (xq), coincides 
with dfu(xo)H restricted to this kernel, 

d 2 F(u)\K^x ) = dfu{x )H\ Ku{xo) . 

Proof of Lemma [7J For xq G M and a control sequence u = (ui, . . . , Ujv) G (int U) N denote 
%i = fui ■ ■ ■ fini^a), i = 1, ■ ■ ■ ,N. Then X{ = / Uj (xj_i) and the image of dF(u) is spanned by the 
vectors 

d d 

fu N ' ' ' fui ■ ■ ■ /ui^o) = d(f UN ■ ■ ■ fu i+ i ) \ x i) Q~ fui [fui-i ' ' ' fui(%o)) 

d 

= dfu^idfu^xo))- 1 ■ ■ ■ (df^ix^t)) l —f Ui (f Ui _ 1 ■ ■ ■ f ui (x )) 
= dfu(x )(Ad Ul ■ ■ ■ Ad Ui _ 1 X+)(x ) = df u (x )Yl(x ). 
This proves the lemma because of the definition of L u {xq) in (J4|). ■ 
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Proof of Lemma [8l Given a control sequence u = (ui, . . . , mat) £ (hit ?7) JV and a vector of 
parameters a = (a 1; . . . , a^r) £ R^, denote 

u\ = ui + (ne, i = l,...,N, 

where e is a small real parameter. Fix xq £ M. In order to find the kernel of the differential dF(u) 
it is enough to find such a = (a%, . . . , a^) that df u * ■ ■ ■ f u \{.xo)/de = 0, at e = 0. We compute 

d N d ( N 

-q-Ju% ■ ■ ■ fu\ (x ) U=o = Yl ai ~^J UN " " /«i " " /«i ( x o) = dfu(x ) I ail^(xo) 

i=l * \i=l / 

where we use the equality established in the preceding proof. Since df a (xo) is an isomorphism, the 
above sum is equal to zero if and only if ^2f =1 aiY£(x ) = 0, which ends the proof. ■ 

Proof of Lemma [91 We will use the notation and the results from the preceding proofs. In 
order to compute d 2 F(u) on the kernel K u (xo) it is enough to compute the second order derivative 
£zfu% • • • fuf (zo)|e=o, for any vector a = (a t , ...,a N ) satisfying Y^f =1 <hY£(x q ) = 0. We have 

d 2 N d d 

^2 /u$, " " " /«? (^o) 1 6=0 = E aia i du~-du~^ UN " ' f Ul ( X °^ 

i,j=l 1 3 

N d ( N d 

= E a ^^E°^/^---/^^) 

N d ( ( N 

= E ai ^u~. [ d fu( x o) ( Yl a i Y £( x °) 

( N N d \ 
= df u (xo) ^^^^-^(^jj , 

where in the last two equalities we use the equality shown in the proof of Lemma [8] and the fact 
that a jYi( x o) = 0. If i < j then, using Proposition [21 we get 

= (Ad„, • • ■ Ad»,_,[X„+, Ad», • •■ Ad^.X+l) (x ) = lY!,Yi](x„) = 2Z| (i„). 



For i = j we have 



Finally, ^r-Yi(xo) = if z > j. Thus, for a £ i^(xo), we obtain 



/ X V 



(d 2 F(u))(a) = -7^fu* N ■ ■ ■ fui{x )\e=o = dfu{x ) I ^ ^ a^Z^ (a; ) I = df u {x )H(a) 



and the proof is complete. ■ 
The proof of Theorem [3] is based on Lemmata [71 El E] and the following result [21 Theorem 20.3]. 
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Lemma 10. Let F : W — > MJ 1 be a map of class C 2 , where W C M fc is an open subset. Fix a point 
w G W and denote the kernel and the image of the differential dF(wo) of F at wq by 

K = kerdF(wo) and L = lmdF(wo). 

If k := codimL > 1 and 

lnd-\d 2 F{w )\ K > k V A G L 1 C (R n )*, A + 0, 
then F(wq) G int F(W). (Here A d 2 F(wo)\K denotes the quadratic form A d 2 F(wo) restricted to K .) 

Proof of Theorem [3]. Consider the map (|T^|) . By the definition of strong local controllabil- 
ity (J3]) it is enough to show that, for a control sequence u = (ui, . . . , ujy), the point F(u) is in the 
interior of the image F(W) of some neighborhood W of u. This is guaranteed if the assumptions 
of Lemma [10] are satisfied, for wq = u. From the hypothesis of Theorem [3] it follows that we can 
use Lemmata [2 M and we see that the assumption of Lemma [10] is fulfilled. This completes the 
proof. ■ 



6 Optimal control 

The general local characteristics of the system, defined at a given initial state Xq and corresponding 
to a given control sequence u (see Section [3]), may also be used for analyzing optimality of the 
control. In this case standard tools from optimization theory are available. Choosing simple 
optimal control problems we indicate the use in these problems of the geometric objects introduced 
earlier: the first variation vector fields and the space L a (xo) spanned by them at Xq, the kernel 
Ku{xq), the second variation vector fields and the corresponding Hessian matrix H. 

Consider the following optimal control problems. Given a system 

S : x(t) =f(x{t-l),u(t)), x(0) = x o , x(t) G M, u(t) G U C M m , 

satisfying conditions (Al)-(A4) and given an initial point x G M, find a control sequence u(t), 
t = 1, . . . , N, which minimizes a function 

(PI) : <p{x(N)). 

The function ip : M — > R, called final cost, is assumed of class C 2 . The number of steps N is assumed 
fixed. Another version of the problem is obtained if, instead, we minimize a cost functional 

N 

(P2): ^(x(iV)) + ^ C (x(t-l),n(t)), 

i=l 

where c : M xlf — > M. is called cost function and assumed of class C 2 . With analogy to continuous- 
time systems we will call (PI) Meyer problem and (P2) Bolza problem. 
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The Bolza problem can be reduced to the Meyer problem by introducing an additional state 
coordinate x°(t) = X^=i c ( x (^ ~ 1), This coordinate satisfies the additional state equation 

x°(t) = x°{t - 1) + c{x(t - 1), u{t)), x°(0) = 0, 

and then problem (P2) is equivalent to problem (PI) with augmented state x = (x°,x) and the 
final cost 0(x(N)) = f(x(N)) + x°(N). 

We will state our results for problem (PI), only, using the notation introduced in Section As 
earlier, we denote u{i) = Ui, x{i) = Xi, and f u (x) = f(x,u). A control sequence u = (u\, . . . , ttjv) 
is called locally optimal for (PI) or (P2) if it is optimal among all sequences in a neighborhood 
U C U N of u. 

Theorem 11. If u is a locally optimal control sequence for problem (PI) and u G [mtU) N then 
the covector 

A = d<f(x N ) df UN (x N -i) df ui (x ) (15) 

satisfies 

\Y£(x o )=0, i = l,...,N, (I) 

and 

XH\ Ku(xo) > 0. (//) 

In condition (II) the inequality means that the quadratic form is non-negative definite. Condi- 
tions (I) and (II) can be called, respectively, first order and second order necessary conditions for 
local optimality. Clearly, (I) is equivalent to A G (L s (x )) ± . A converse result is more complicated. 

Theorem 12. Given an initial point xq and a control u G (int U) , let A be the covector defined 
by ()15p . Assume that (I) holds and condition (II) is strengthened to 

\H\ Ku{xo) > 0. (Ill) 

Then there exists a quadratic form Q on the image L C T XN M of the composed map 

dfu N ( x N-i) dfui(x Q ) 

such that if 

d 2 y{x N )\ L > Q (IV) 
then u is locally optimal for problem (PI). Here Q depends on system E and xq, u and dip(xN)- 

Remark 13. Condition (IV) means that the quadratic form d 2 ip(x^)\L — Q is positive definite. 
The form Q will be determined by formulae ( 12"2"|) . f l2U|) in the proof. Note that if (IV) does not hold 
for a given (p then it is satisfied for another if with the same dip(xN) and suitable d 2 ip(xN)- 
Remark 14. Clearly, the problem (PI) can be reduced to a standard optimization problem. Then 
standard second order conditions for optimality can be used, as will be seen at the beginning of 
the proof of Theorem [TTJ However, these conditions are impractical in use as they involve multiple 
compositions of nonlinear maps. Neither they give much geometric insight into the problem. 
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Proof of Theorem [5]. The assumptions of the theorem imply that there is a covector A at 
x G M such that conditions (I) and XH\ Ku r X0 ) > hold. Define a covector A at xjy = f UN ■ • • fui{ x o), 

A = A (^(xq))" 1 ■ ■ • (^/^(xjv-i)) -1 , 

and the linear function <p(x) = Xx. Then Xq, u and ip satisfy assumptions (I) and (III) in Theorem 
[T21 Let Q be a symmetric matrix such that Q\l satisfies condition (IV) in Theorem [T21 Define 
another function ^ 

(p(x) = Xx + -(x - x N ) T (Q + el){x - x N ), 

where I is identity matrix and e > 0. Then satisfies condition (IV), too, (with Q replaced by Q) 
and Theorem [T2] implies that control u is locally optimal, for problem (PI) with final cost (p. 

The function <p has regular level sets in a neighborhood of x^. Since optimality of u implies 
that the whole local reachable set from xq (obtained using controls v in a neighborhood of u) lies 
above or on the level set of the minimal value, no neighborhood of xn is covered by the local N-step 
reachable set. Thus £ is not strongly (xo, ■uj-locally controllable. ■ 

Proof of Theorem llll We use the notation introduced in Section formula (JSJ), and assume 
that a local coordinate system is chosen in a neighborhood of x^ G M. We have 

x N = F(u) = fu(xo), 

where /„ is the composition f u = f UN ■ ■ ■ f Ul . Clearly, problem (PI) is equivalent to minimization 
of the composed function 

ip(u) = ip o F(u). 

Since dip{u) = d(p(x^) dF(u), the first order necessary condition for minimum can be written as 

dtp{x N ) dF{u) = (16) 

and the second order condition 

d 2 ip(u) = dip{x N ) d 2 F(u) + d 2 ip{x N ){dF{u) ■ ,dF{u) ■ ) > 0. (17) 

Above we treat d 2 F(u) as a vector-valued symmetric bilinear form and d 2 p){x^) as a symmetric 
bilinear form. The inequality means that the corresponding quadratic form is non-negative definite. 
If the above quadratic form is restricted to the kernel of dF(u), equal to the space K u (xq) by 
Lemma [8], then the second term vanishes and we get the condition 

d<p(x N )d 2 F{u)\ Ku{xo) > 0. (18) 

We will show that conditions ffTBT) and (jTSJ) are equivalent to assertions (I) and (II) of the theorem. 
We first prove equivalence of conditions (TT6"j) and (I). Note that (I16p can be written in the form 

dip{x N )lmdF{u) = 0. (19) 
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Since the covector A in ([15]) is equal to A = dip(xN) dfu(xo) and lmdF(u) = dfu{xo)Lu{xo), by 
Lemma we can write f[T$j) as 

XL a (x ) = 0, 

which is equivalent to condition (I). 

The equality d 2 F{u)\ K -^ X0 ) = df u (x )H\ K -( X0 ) from Lemma [9] implies that condition (fT8j) is 
equivalent to 

XH\ Ku(xo) > (20) 

which is condition (II) in the theorem. ■ 

Proof of Theorem 1121 We use the notation from the above proof. Notice that condition (TTB"]) 
and condition (fT7|) strengthened to strong inequality (positive defmiteness) are standard sufficient 
conditions for local optimality of the point u, for the function ip. Thus it is enough to show that 
ffTB"]) and the strengthened version of f[TT|) follow from the assumptions of the theorem. 

We have seen in the first part of the proof of Theorem [TTJ that assumption (I) is equivalent to 
condition ( TIB"]) . Thus we should prove that, with appropriately chosen d 2 (p(xN), the strengthened 
version of ( II 7p holds. This means that, for any nonzero vector v, we should have 

d<p(x N )d 2 F(u)(v,v) + d 2 <p(x N )(dF(u)v,dF(u)v) > 0. (21) 

In the standard basis in M. mN we can identify the bilinear forms appearing in ( 12T|) with symmetric 
matrices 

A ~ d(f{x N ) d 2 F{u), B ~ d 2 (f{x N ){dF{u) ■ , dF(u) ■ ). 

Denote for brevity K = Ku(xq). Define the subspace kerA = {v G M m7V : Av = 0} C M. mN and 
notice that ker A D K — {0}. This follows from condition (III). Namely, the quadratic form A 
restricted to K is equal to XH\ K , by Lemma |H] and the definition (fT5"j) of A. Thus A\ K is positive 
definite. Since A restricted to ker A is zero, the intersection of ker A and K must be trivial. 

We will bring the bilinear form A to a block-diagonal form. Since ker A PI K — {0}, we can 
choose a complement E of ker A in W mN so that K C E. Since A is nondegenerate on E and it is 
positive definite on K, the A-orthogonal complement K 1 - — {y G E : v T Aw = 0VtuG K} of in 
E has trivial intersection with K. Thus, E = K © i^ -1 and R mJV is the direct sum 

R mN = K®K L ©ker A 

From the definitions of ker A and K x it follows that, in this decomposition, 





(An 














\ 


A = 





A 22 










-B22 


B23 




V o 











-B32 





where the zeros in B appear because v T Bw = if one of the vectors v, w belongs to K = ker dF(u). 

From the definition of B and the fact that dF(u) is injective on the complement K ± © ker A 
of K = ker dF(u) we deduce that the blocks B^ can be chosen arbitrarily, with an appropriate 
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choice of d 2 (p(x^)- Our aim is to characterize those d 2 (p(xN) which make B satisfying (j2ip . that is, 
A + B > 0. 

The weaker condition A + B > is satisfied, if we choose B = B with B 22 = —A 22 and the 
remaining B^ = 0, since An = A\% = \H\k > 0, as proved earlier. Such matrix B is obtained 
with choosing <p = (p so that d 2 (p(x^) = Q, where Q is a symmetric matrix characterized via the 
equality 

S T QS = B (22) 
and S is the matrix of the linear map dF(u) : IR mAr — > T XN M, in a linear basis in T XN M. 
Now we define 

Q = Qk (23) 

that is, Q is the quadratic form determined by the matrix Q in fl22|) . restricted to the subspace 
L = lmdF(u) = ImS, and 9? is chosen so that d 2 (p(x]y) \l > Q- Then we have A + B > 0, where 
£> = S T d 2 ip(x]y)S. Indeed, then S T d 2 ip(x]y)S > S T QS (since the map defined by S is injective) 
and A + B is block-diagonal, with two positive definite blocks, thus A + B > 0. ■ 

Remark 15. It follows from the above proof that the quadratic form Q defined by fl22|) . fl23l) has 
the property that for any other quadratic form Q' having the property in the theorem we have 
Q' ^ Q- The form Q is uniquely defined by this minimality property. 

Example 16. Consider the system given by the dynamics ( TTUj) and a final cost function ip : M — > R. 
Then, for any initial condition, a control sequence u = (ui, u 2 , U3, u&) such that Ui 7^ for all i can 
not be optimal for problem (PI) if only dip(xN) 7^ at the final point Xjy. To prove suppose that 
u is optimal. Then the covector A given by (ITS!) satisfies condition (I), thus A G (^^(x))^. We 
have checked in Section H] (Case I) that either dimL s (x) = 3 (then A = and d(p(x^) = 0), or 
dimL s (x) = 2 and then any A G (L s (x)) ± is unique up to a multiplier, thus it can be taken as in 
formula fTl5|) . It was checked in Section H] that, for such nonzero A, the quadratic form Aif|^-( x ) is 
indefinite. Thus the necessary condition (III) in Theorem [TT] is not satisfied and we conclude that 
u is not an optimal control, if dip(xN) 7^ 0. 

Cases Ha and lib in the example in Section H] admit optimal control sequences. This can be 
seen from Theorem [121 Namely, in Case Ha we have codim L s (x) = 1 and covectors A G (L s (x)) ± 
(equivalently A satisfying condition (I)) are unique, up to multiplier. Let d(p(xN) be such that the 
corresponding A given by f TTS]) satisfies the first order condition (I). Then A G (L^x))- 1 and we 
have checked that, for such A, the quadratic form XH restricted to the kernel is positive definite, 
thus it satisfies the sufficiency condition (III). From Theorem [121 we deduce that the control u is 
optimal, provided that d 2 <p(xN) > Q, for some symmetric matrix Q depending on it and the initial 
state (this matrix is determined from condition ( 122]) ). In fact, it is enough that d 2 (p(x]v)\L > Q\l, 
where L = lmdF(u). The conclusion in Case lib is similar. 
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7 Optimality conditions using Hamiltonian 



We will now state second order necessary conditions for optimal control problems applying a for- 
malism similar to Hamiltonian formalism used for continuous-time systems. In order to be able to 
use duality of vectors and covectors we assume that M is an open subset of an affine or a vector 
space. In this case the difference x(t) — x(t — 1) of two consecutive states can be treated as a vector. 

For simplicity we will assume that M is an open subset of a vector space, 

M C R n . 

In this case, we can canonically identify all the spaces of (tangent) vectors so that T X M = R n , for 
any x G M, and similarly identify the spaces of covectors, T*M = (R n )*. 

As earlier, we consider an initialized system £ : x(t) = f(x(t — l),u(t)), x(0) = Xq. We define 
the Hamiltonian H : M x (R n )* x U -»> R of £, 

H(x,p,u) = pf(x,u), 

where we treat p as a linear function acting on v = f(x,u), in coordinates pv = YliPi vt - 
Remark 17. If M is an open subset of an affine space then a more natural Hamiltonian is 

H(x,p,u) = p(f(x,u) - x), 

since v = f(x, u) — x is a vector and pv is well defined by duality of covectors and vectors. Further 
considerations hold with both definitions, with necessary modifications stated in remarks. 

In order to state the second order optimality conditions in terms of the Hamiltonian we consider a 
control sequence u = (u±, . . . , un), where u t = u(t), t = 1, . . . , N, and the corresponding trajectory 
(x , . . . , xn), x t = x(t). We can restrict the control sequence to its initial part 

u l = Oi, . . .,Ui), 

for any fixed i = 1, . . . , N. All the definitions from the previous sections work if the sequence u is 
replaced with the restricted sequence u\ In particular, using definitions from Section [31 we denote 
the vector fields and the Hessian matrix corresponding to the restricted sequence by 

L l( x o) ■= L u t(x ), Kl(x Q ) := K U i(x ), HI := 

Then 

4(*o)c---C^(* ) 

and 

where, in the latter case, we use the natural embeddings 

R m c R m x R m C C R m x N • t - mes xR m 
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given by (wi, . . . , m) ^ (u h . . . , Ui, 0, . . . , 0). 

Consider again the optimality problem (PI) from Section [6j for system E satisfying (Al)-(A4). 
Theorem [TT] can be reformulated in terms of the Hamiltonian in the following way. 

Theorem 18. If (u(l), . . . ,u(N)) £ (int U) N is an optimal control sequence for problem (PI) and 
x(0), . . . , x(N) is the corresponding trajectory, then there exists a sequence of covectors p(0) , . . . ,p(N), 
with p(N) = d(p(xjf), such that the following conditions are satisfied. The state equations hold 

OH 

<t) = — (x(t - l),p(t),u(t)), t=l,...,N, (E) 
together with the adjoint equations 

dH 

p(t - 1) = —(x(t - l),p(t), u(t)), t = l,...,N, (E*) 

the criticality condition 

—{x{t - l),p{t),u(t)) = 0, t — 1, . . . , N, (CC) 

and the second order necessary condition 

p(0)Hl\ KUxo) > 0, t = l,...,N. (SO) 

Remark 19. Note that equations (E) together with (E*) are equivalent to 

x(t) = f(x(t-l),u(t)), t = l,...,N, (E) 

df 

pit - 1) = P(t)^(x(t - 1), u(t)), t = 1, . . . , N. (E*) 

Remark 20. In the case of M being a subset of an affine space and the Hamiltonian of the form 
H(x,p, u) = p(f(x, u) — x) the theorem holds with the modified state equations 

dH 

x(t) - x(t - 1) = — (x(t - 1), u(t)), t = l,...,N, (E) 
and the adjoint equations 

dH 

p(t-l)-p(t) = —(x(t-l),p(t),u(t)), t = l,...,N, (E*) 

Proof of Theorem 1181 Let u(l) = u±, . . . ,u(N) = un be an optimal control sequence and 
x(l) = X\, . . . ,x(N) = xn the corresponding trajectory starting from x(0) = Xq. By Theorem [TT] 
the covector 

A = d(p(x N ) df UN (x N _i) df ui (x ) 

satisfies Ay^(xo) = for i = 1, . . . , N (equivalently, A £ (L^xo)) -1 )) and XHu\kh(x ) — 0- 
Denoting p(i) = pi we put 

p N = A (dfu^xo))- 1 ■ ■ ■ (df^ixN-i))- 1 
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and consider the sequence 



Pt = PN<lf UN (x N -i) ■ ■ ■ df Ut+1 (x t ) = A (<i/ Ul (xo)) _1 ■ • • (dfuXxt-i))' 1 , 

t = 1, . . . , iV — 1 (for t — the latter equation does not hold and will not be used). Note that 

Po = A and pn = dtp(xN)- 

The above sequence satisfies p t -\ = ptdf Ut (x t -\) = dH/dx(x t -i,pt, u t ) which is the adjoint equation 
(E*). We claim that this solution satisfies the other assertions of the theorem, too. The equation 
(E) is satisfied by the definitions of the sequences u(t), x(t) and p{t) and Remark [T§1 

We will show that (CC) follows from condition XY^(xq) — 0. Namely, for i = 1, 

dfu 

= XY^(xo) = AX+(x ) = Po(df Ul (xo)y l -^-(xQ)\ u=Ul 
df U/ v., dH 

= P1-q-[Xq)\ u=Ui = — (Xq,Pi,Ux). 

For i = 2, . . . , N we have 

= A^(x ) = A(Ad Ul ---Ad Ui _ 1 X+)(x ) 
= A {df Ul (x Q )Y l ■ ■ ■ (£(f tt ._ 1 (a; i _ 2 ))- 1 X+(a; i _i) 

= Pi_iXj(xi_i) =Pi-i(# Ml (xi_i)) _1 ^(xi_i)| u=n8 

Pi \%i—l)\u=Ui \Xi— 1) Pi; ">i) > 

and we see that condition (CC) is satisfied. 

In order to show that condition (SO) holds note first that it holds for t = N, by Theorem [TT1 
Namely, with our construction of the adjoint sequence p(N), . . . ,p(0) we have po = p(0) = A, where 
A is as in Theorem [TTJ We claim that for 1 < t < N condition (SO) follows from (SO) satisfied 
for t = N. This is rather obvious. Namely, under the natural embedding W m C M. Nm defined by 
(ui, . . . ,Ui) i — y (ui, . . . , Ui, . . . , 0), we have K^(x ) C K^(x ) which follows from the definition 
of K % u (xo). The matrix H % u forms the upper-left block of , of size im, corresponding to the 
subspace M. tm C M. Nm . Thus positive definiteness of the whole matrix implies the same property of 
the sub-matrix. The proof is complete. ■ 

Second order necessary conditions for geometric optimality can also be stated using Hamiltonian. 

Theorem 21. If u = (w(l), . . . , u(N)) G (mtU) N is a geometrically optimal control sequence of 
an initialized invertible system E and x(l), . . . ,x(N) is the corresponding trajectory starting from 
x(0) = xq, then dimL u (xo) < n and, for any nonzero A G (Lu(xo)) ± , there exists a sequence 
of nonzero covectors p(0) , . . . , p(N) with p(0) = A such that the state equations (EJ, the adjoint 
equations (E*), the criticality condition (CC) and the following index condition 

Ind-(XHt)\ Kh(xo) <codim4(x ), t = l,...,N, (IC) 
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hold. In particular, (IC) implies that if codimL^(xo) = for some t 6 {1,...,N} then the 
trajectory is not geometrically optimal. If codimL^(xo) = 1, then (IC) means that the quadratic 
form (A Ht) | K t / XQ ^ is non-negative definite. 

Proof of Theorem 1211 The proof is analogous to the proof of Theorem [TBI with the difference 
that one should use Theorem [HI instead of Theorem [TTJ, for determining the covector A with required 
properties. We leave the details to the reader. ■ 
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